Study shrinkage and DART in xgboost modeling using a simple dataset

Machine Learning
A small R experiment that illustrates how shrinkage and DART dropout change the behavior of XGBoost trees on a simple toy dataset.
Author

Yang Liu

Published

November 15, 2018

A small dataset is often the best way to understand a modeling algorithm. Inspired by my colleague Kodi’s excellent work showing how xgboost handles missing values, I used a simple 5x2 dataset to show how shrinkage and DART influence tree growth in an XGBoost model.

Data 

set.seed(123)
n0 <- 5
X <-  data.frame(x1 = runif(n0), x2 = runif(n0))
Y <-  c(1, 5, 20, 50, 100)
cbind(X, Y)
##       x1      x2   Y
## 1 0.2876 0.04556   1
## 2 0.7883 0.52811   5
## 3 0.4090 0.89242  20
## 4 0.8830 0.55144  50
## 5 0.9405 0.45661 100

Shrinkage

  1. Step-size shrinkage is a major tool for preventing overfitting, or over-specialization.
  • The R documentation says that the learning rate eta has range [0, 1], but xgboost takes any value of (eta). Here I select eta = 2, and the model can perfectly predict in two steps: the train RMSE from iteration 2 is 0, and only two trees are used.
    Of course, it is a bad idea to use a very large eta in real applications as the tree will not be helpful and the predicted value will be very wrong.
  • The max_depth is the maximum depth of a tree, I set 10 but it won’t be reached
  • By default, there is no other regularization
  • By setting base_score = 0 in xgboost we can add up the values in the leaves of two trees to get every number in Y: 1, 5, 20, 50, 100 
# non-zero skip_drop has higher priority than rate_drop or one_drop
param_gbtree <- list(objective = 'reg:linear', nrounds = 3, 
                   eta = 2,
                   max_depth = 10,
                   min_child_weight = 0,
                   booster = 'gbtree'
)

simple.xgb.output <- function(param,...){
  set.seed(1234)
  m = xgboost(data = as.matrix(X), label = Y, params = param,
              nrounds = param$nround, 
              base_score = 0)
  cat('Evaluation log showing testing error:\n')
  print(m$evaluation_log)
  pred <- predict(m, as.matrix(X), ntreelimit = param$nrounds)
  cat('Predicted values of Y: \n')
  print(pred)
  pred2 <- predict(m, as.matrix(X), predcontrib = TRUE)
  cat("SHAP value for X: \n")
  print(pred2)
  p <- xgb.plot.tree(model = m)
  p
}
simple.xgb.output(param_gbtree)
## [1]  train-rmse:22.360680 
## [2]  train-rmse:0.000000 
## [3]  train-rmse:0.000000 
## Evaluation log showing testing error:
##    iter train_rmse
## 1:    1      22.36
## 2:    2       0.00
## 3:    3       0.00
## Predicted values of Y: 
## [1]   1   5  20  50 100
## SHAP value for X: 
##          x1     x2 BIAS
## [1,] -29.33 -4.867 35.2
## [2,] -26.00 -4.200 35.2
## [3,] -24.93  9.733 35.2
## [4,]  16.50 -1.700 35.2
## [5,]  66.50 -1.700 35.2
  • If eta = 1
    Then no perfect prediction can be made, and the trees grow in a more conservative manner. 
# non-zero skip_drop has higher priority than rate_drop or one_drop
param_gbtree <- list(objective = 'reg:linear', nrounds = 3, 
                   eta = 1,
                   max_depth = 10,
                   min_child_weight = 0,
                   booster = 'gbtree'
)
simple.xgb.output(param_gbtree)
## [1]  train-rmse:22.831995 
## [2]  train-rmse:11.415998 
## [3]  train-rmse:5.707999 
## Evaluation log showing testing error:
##    iter train_rmse
## 1:    1     22.832
## 2:    2     11.416
## 3:    3      5.708
## Predicted values of Y: 
## [1]  0.875  4.375 17.500 50.000 87.500
## SHAP value for X: 
##          x1     x2  BIAS
## [1,] -27.18 -3.999 32.05
## [2,] -24.20 -3.478 32.05
## [3,] -24.11  9.564 32.05
## [4,]  20.41 -2.463 32.05
## [5,]  56.98 -1.525 32.05

DART: Dropout - MART

DART (paper in JMLR) adapts dropout from neural networks to boosted regression trees, or MART: Multiple Additive Regression Trees. DART aims to further prevent over-specialization. In xgboost, use booster = ‘dart’ and tune the related hyperparameters (R documentation).

skip_drop

  • The skip_drop(default = 0, range [0, 1]) is the probability of skipping dropout. It has a higher priority than other DART parameters. If skip_drop = 1, the dropout procedure would be skipped and dart is the same as gbtree. The setting below gives the same result as the gbtree above (results omitted): 
param_gbtree <- list(objective = 'reg:linear', nrounds = 3, 
                   eta = 2,
                   max_depth = 10,
                   booster = 'dart',
                   skip_drop = 1,  # = 1 means always skip, = gbtree
                   rate_drop = 1,  # doesn't matter since drop is always skipped
                   one_drop = 1
)

rate_drop

  • If skip_drop(), rate_drop (default = 0, range [0, 1]) will drop a fraction of the trees before the model update in every iteration.
  • The DART paper JMLR said the dropout makes DART between gbtree and random forest: “If no tree is dropped, DART is the same as MART (gbtree); if all the trees are dropped, DART is no different than random forest.”
  • If rate_drop = 1 then all the trees are dropped, a random forest of trees is built. In our case of a very simple dataset, the ‘random forest’ just repeats the same tree nrounds times: 
param_dart1 <- list(objective = 'reg:linear', nrounds = 3, 
                   eta = 2,
                   max_depth = 10,
                   booster = 'dart',
                   skip_drop = 0,  
                   rate_drop = 1,  # doesn't matter since drop is always skipped
                   one_drop = 1
)
simple.xgb.output(param_dart1)
## [1]  train-rmse:22.360680 
## [2]  train-rmse:16.948286 
## [3]  train-rmse:19.388212 
## Evaluation log showing testing error:
##    iter train_rmse
## 1:    1      22.36
## 2:    2      16.95
## 3:    3      19.39
## Predicted values of Y: 
## [1]  0.5833  2.9167 11.6667 58.3333 58.3333
## SHAP value for X: 
##          x1      x2  BIAS
## [1,] -22.94 -2.8389 26.37
## [2,] -21.00 -2.4500 26.37
## [3,] -20.38  5.6778 26.37
## [4,]  32.96 -0.9917 26.37
## [5,]  32.96 -0.9917 26.37

one_drop

  • If one_drop = 1 then at least one tree is always dropped. If I let rate_drop=0, but one_drop = 1, the dropping was still working, and the trees were built in a more conservative manner. Since the first tree will be dropped, the second tree is the same as the first one 
param_dart2 <- list(objective = 'reg:linear', nrounds = 5, 
                   eta = 2,
                   max_depth = 10,
                   booster = 'dart',
                   skip_drop = 0,  
                   rate_drop = 0,  # doesn't matter since drop is always skipped
                   one_drop = 1
)
simple.xgb.output(param_dart2)
## [1]  train-rmse:22.360680 
## [2]  train-rmse:16.948286 
## [3]  train-rmse:15.066435 
## [4]  train-rmse:15.237643 
## [5]  train-rmse:14.225216 
## Evaluation log showing testing error:
##    iter train_rmse
## 1:    1      22.36
## 2:    2      16.95
## 3:    3      15.07
## 4:    4      15.24
## 5:    5      14.23
## Predicted values of Y: 
## [1]  0.7037  5.8848 14.7737 53.2510 68.8066
## SHAP value for X: 
##          x1     x2  BIAS
## [1,] -24.09 -3.893 28.68
## [2,] -19.28 -3.522 28.68
## [3,] -18.88  4.974 28.68
## [4,]  22.36  2.211 28.68
## [5,]  41.70 -1.578 28.68
  • Similar conservative effect if I set skip_drop to be non-zero: 
param_dart3 <- list(objective = 'reg:linear', nrounds = 5, 
                   eta = 2,
                   max_depth = 10,
                   booster = 'dart',
                   skip_drop = 0,  
                   rate_drop = 0.5,  # doesn't matter since drop is always skipped
                   one_drop = 0
)
simple.xgb.output(param_dart3)
## [1]  train-rmse:22.360680 
## [2]  train-rmse:0.000000 
## [3]  train-rmse:7.453556 
## [4]  train-rmse:9.316948 
## [5]  train-rmse:9.823564 
## Evaluation log showing testing error:
##    iter train_rmse
## 1:    1     22.361
## 2:    2      0.000
## 3:    3      7.454
## 4:    4      9.317
## 5:    5      9.824
## Predicted values of Y: 
## [1]  0.8333  4.1667 16.6667 63.8889 83.3333
## SHAP value for X: 
##          x1     x2  BIAS
## [1,] -28.89 -4.056 33.78
## [2,] -26.11 -3.500 33.78
## [3,] -25.22  8.111 33.78
## [4,]  31.53 -1.417 33.78
## [5,]  50.97 -1.417 33.78

Letting one_drop = 1 also gives result more conservative, and smaller train-rmse if use same rounds.